Papers with Massive Multitask Language Understanding
Confidence-Driven Multi-Scale Model Selection for Cost-Efficient Inference (2026.findings-eacl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have revolutionized inference across diverse natural language tasks, with larger models performing better but at higher computational costs. |
| Approach: | They propose a confidence-driven strategy that dynamically selects the most suitable model based on confidence estimates. |
| Outcome: | The proposed approach reduces token usage by approximately 60% and improves cost efficiency on the Massive Multitask Language Understanding (MMLU) benchmark. |
Are We Done with MMLU? (2025.naacl-long)
Copied to clipboard
Aryo Pradipta Gema, Joshua Ong Jun Leang, Giwon Hong, Alessio Devoto, Alberto Carlo Maria Mancino, Rohit Saxena, Xuanli He, Yu Zhao, Xiaotang Du, Mohammad Reza Ghasemi Madani, Claire Barale, Robert McHardy, Joshua Harris, Jean Kaddour, Emile Van Krieken, Pasquale Minervini
| Challenge: | MMLU is widely adopted but its ground truth errors obscure the true capabilities of LLMs. |
| Approach: | They propose a framework for identifying dataset errors using a novel error annotation protocol and a subset of 5,700 manually re-annotated questions. |
| Outcome: | The proposed framework is based on 5,700 re-annotated questions from the MMLU benchmark. |
MMLU-CF: A Contamination-free Multi-task Language Understanding Benchmark (2025.acl-long)
Copied to clipboard
Qihao Zhao, Yangyu Huang, Tengchao Lv, Lei Cui, Qinzheng Sun, Shaoguang Mao, Xin Zhang, Ying Xin, Qiufeng Yin, Scarlett Li, Furu Wei
| Challenge: | Multiple-choice question datasets like Massive Multitask Language Understanding (MMLU) have inevitably led to benchmark contamination, resulting in unreliable evaluation. |
| Approach: | They propose a contamination-free MCQ benchmark called MMLU-CF which reassesses LLMs’ understanding of world knowledge by averting both unintentional and malicious data contamination. |
| Outcome: | The proposed MMLU-CF reassesses LLMs’ understanding of world knowledge by averting both unintentional and malicious data contamination. |